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ABSTRACT 

Educators seek to develop 21st century skills in the classroom by incorporating educational materials other than textbooks into 
their lessons, such as digitally available activities, videos, and visualizations. A problem that educators face is that no review 
process similar to the formal adoption processes used for K-12 textbooks or the college-textbook review process exists for 
these types of online educational resources. However, educators need authoritative high-quality digital teaching materials. 
The scientific journal peer-review system offers a well-established model to adapt to the requirements of a peer review of 
educational materials. In this paper, we review ten review processes developed to evaluate digital geoscience educational 
resources and focus in detail on a rigorous iterative peer-review process recently developed by the Climate Literacy and 
Energy Awareness Network (CLEAN) project. This process builds upon existing efforts and emphasizes the "curation" of a 
digital collection that addresses the Essential Principles of Climate Literacy and the Energy Literacy Principles. Providing 
educators with thoroughly reviewed educational materials is especially important for fast changing, societally important, and 
sensitive areas such as climate and energy science. © 2012 National Association of Geoscience Teachers. [DOI: 10.5408/12-324.1] 
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INTRODUCTION 

The peer-review process for scientific journals is well 
established. In contrast, review of educational material is not 
as formally developed. For example, only half of all U.S. states 
follow a formal adoption process for K-12 textbooks (Phillips, 
2011). The problem is particularly acute for digital educational 
materials such as online lesson plans and videos, where only a 
few U.S. states have implemented a system to review these 
materials. However, the need for authoritative and effective 
digital educational materials is increasing, as educators seek to 
teach 21st-century skills (Borgman et al., 2008), expose 
students to new media, introduce variety into the instructional 
approach, and include the latest scientific findings in lessons. 

The transition from printed educational materials to new 
media and cyberlearning poses numerous challenges to 
educators, some of which apply more readily to K-12 
educators than to college faculty: First, educators may lack 
knowledge and training in how to find, use, and adopt 
digital materials (Hanson and Carlson, 2005; McMartin et al., 
2008; Mervis, 2009a). Second, the validity of resources can be 
difficult to assess (Hanson and Carlson, 2005; Cafolla, 2006). 
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Third, professional development to support the use of these 
resources in the classroom is not always available (Hanson 
and Carlson, 2005; Johnson, 2011; Buhr et al., 2012). Fourth, 
there may be a lack of sufficient technological institutional 
support, such as availability of computers, software, or 
download restrictions (McMartin et al., 2008). Fifth, resource 
content may not be aligned with state and national 
standards, an important criterion for K-12 educators 
(Hanson and Carlson, 2005; Mervis, 2009a; Bangay and 
Blum, 2010; Wise, 2010; Lynds, 2011). In this paper, we 
review efforts to support educators in the use of digital 
educational materials, with a particular focus on the solution 
of the first two challenges described above. 

One way to overcome the problem of accessing reliable 
and high-quality digital teaching materials is through 
curated collections of digital resource or digital libraries of 
educational materials. Digital libraries or collections present 
resources, otherwise scattered over the Internet, together in 
a coherent and accessible portal with easy search interfaces 
(Hanson and Carlson, 2005; Lempinen, 2006). Prominent 
examples are the collections within the National Science 
Digital Library (NSDL), the Science Education Resource 
Center (SERC), and the Multimedia Educational Resource 
for Learning and Online Teaching (MERLOT) as well as 
commercial efforts such as lessonplanet.com, tutor.com, 
net.trekker, learner.com, and Brokers of Expertise. 

The introduction of review systems comparable to the 
scholarly journal review system has been attempted by 
numerous groups to address the described barrier of variable 
and unreliable scientific and educational quality of digital 
resources. Ten prominent examples of review processes for 
educational material are summarized in Table I. 

Providing educators with authoritative and scientifically 
accurate teaching resources is especially important for publicly 
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sensitive subjects including evolution, climate change, and 
energy, which can lead to controversy and confusion in the 
classroom. In this paper, we focus on climate and energy 
topics. How can an educator who may lack knowledge of the 
underlying science know that an online resource is scientif¬ 
ically accurate, pedagogically useful, and technically robust? 
Educators have reported being challenged by parents or 
colleagues when teaching climate science or they opt to teach 
"both sides" (Wise, 2010; Chasteen, 2011; Johnson, 2011; 
Buhr et al., 2012). Educators have described a need for high- 
quality teaching resources in this field (Lynds, 2011; Buhr et 
al., 2012). The Climate Literacy and Energy Awareness 
Network (CLEAN) addresses these challenges by providing 
educators with a collection of rigorously reviewed digital 
educational materials on these topics. 

In this paper, we briefly summarize the scholarly journal 
peer-review system as well as efforts to modify this 
established process in the evolving field of educational 
material review. We then describe existing efforts and 
previous attempts to build review systems for digital 
collections of teaching materials in the geosciences. Finally, 
we present the review system developed by the CLEAN 
project team and describe how we evolved the educational 
and scientific review processes from previous efforts. 

PEER-REVIEW PROCESS 

The Scholarly Peer-Review Process 

Peer review is a fundamental pillar of the scientific 
process and thus of creation and dissemination of knowl¬ 
edge (Hovav and Gray, 2006; Mandviwalla et al., 2008; 
Harley and Acord, 2011). Its beginnings date back to the 
17th century (Kronick, 1990; Mulligan, 2004). The process is 
well-established for scientific journals and generally follows 
the model in Fig. 1. The scientific community only accepts 
new scientific findings or theories after publication in peer- 
reviewed journals (Mulligan, 2004; Suls and Martin, 2009), 
which also affects tenure decisions in academic careers 
(Speier et al., 1999; Siemens, 2001). Therefore, most 
scientists are intimately familiar with the process both as 
submitters and reviewers of scientific articles. Criticism of 
the peer-review process includes the following: It is a slow 
and expensive process; it lacks inter-reviewer reliability, and 
there is the possibility that the reviewers may be biased, may 
abuse the peer review for personal career enhancement, or 
steal ideas (Harnad, 1998; Moller and Jennions, 2001; 
Mulligan, 2004; Oxman and Guyatt, 2006; Smith, 2006; 
Grainger, 2007; Mandviwalla et al., 2008; Suls and Martin, 
2009; Harley and Acord, 2011). 

Modifications to Scholarly Peer-Review Process in the 
Digital Age 

Scholarly communication has shifted from largely print- 
based journals to electronic journals (often with a print 
counterpart) within the last two decades (Tenopir, 2004; 
Greco et al., 2006; Moghaddam, 2007; Rowlands, 2007). 
Electronic journals are attractive because they are easy to 
access, original data is often provided as online supplements, 
and publication follows quickly after the review process 
(Liew et al., 2000; Siemens, 2001; Hovav and Gray, 2006). 
With increasing numbers of electronic journals, alternate 


models to the traditional peer-review process have been 
explored by publishers, including 1) mediated open discus¬ 
sions prior to publication, 2) publication of accepted 
manuscript along with comments by reviewers to increase 
accountability, and 3) tentative publication of accepted 
manuscript along with peer reviews for open public 
comments that can be included before publication (Harnad, 
1998; Sumner et al., 2000; Weller, 2000; Mizzaro 2003; 
Mulligan, 2004; Hovav and Gray, 2006; Smith 2006; Suls and 
Martin, 2009; Campion and Drazen, 2010). None of these 
modified models has been widely implemented. 

Review Process for Educational Materials 

The process of rigorous peer review of educational 
resources has not been applied consistently and is not as 
widely accepted in the educational as in the scientific 
community (Eibeck, 1996; Knight et al., 2004). One reason 
is that curriculum developers are not producing new cutting- 
edge content; instead, they repackage authoritative, peer- 
reviewed scientific knowledge to make it accessible for the 
target audience (Fincher and Work, 2005). However, 
repackaging and simplifying established scientific knowl¬ 
edge for a specific audience is nontrivial because oversim¬ 
plification might lead to the development of misconceptions 
among students. Thus, a review process to ensure a correct 
translation of scientific findings into effective teaching 
materials is crucial. The journal peer-review process is an 
established model that can inform the development of a 
review process for educational materials. 

Reviewing educational resources poses additional chal¬ 
lenges to the review process as compared to the process 
established for scholarly communication. Educational mate¬ 
rials have to be scientifically accurate and current, pedagog¬ 
ically effective, presented in an attractive format, and designed 
in compliance with effective educational practices (Eibeck, 
1996; Ruiz et al., 2007). If published electronically, the 
reviewers also have to evaluate the technical quality and 
feasibility as well as usability of the content (Weatherley et al., 
2002; Knight et al., 2004; Kastens, 2005). A single reviewer is 
rarely qualified to assess all of these aspects; thus, a panel of 
experts is typically required to review educational materials. 

Review processes for textbooks exist but are different for 
textbooks used at the college level than for K-12 textbooks. 
Publishers of college-level textbooks send the textbooks out 
for review to scientific experts in the field before publication, 
leaving the authority of the review in the publishers' hands. 
In contrast, the K-12 textbook review and adoption process 
is often regulated by the states. 

In the United States, 22 states have established review 
processes to approve or adopt instructional materials, mainly 
textbooks (Texas Education Agency, 2010; Phillips, 2011). 
These review processes are implemented by states' individ¬ 
ual boards of education (Finn and Ravitcb, 2004; Phillips, 
2011). However, California and Texas, which have the 
largest state populations, traditionally set standards for the 
adoption process to ensure quality, and materials that meet 
these standards are frequently adopted by other states. For 
example, in California the Curriculum Commission appoints 
instructional material reviewers as well as content review 
experts. Each reviewer submits an independent review; the 
publisher and the public can comment before a final decision 
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regarding adoption of the materials is made (California 
Department of Education, 2009). Thus, the established 
review process for educational resources is the effort of a 
multidisciplinary team, requiring multiple years. 

The unlimited and inexpensive digital publishing op¬ 
portunities of educational resources of different granularities 
(digital textbooks and full curriculum to single animations 
and visualizations) challenge this established process 
(CLRN, 2008; Mardis et al., 2010). Digital educational 
materials allow much faster publication than complete print 
textbooks. The short turnaround is particularly important for 
evolving scientific fields where an adoption process of 
multiple years may result in materials being already outdated 
when published (Cafolla, 2006). However, the fast process of 
digital publication and the absence of an accepted review 
process make it more difficult for the educators to judge the 
quality and authority of a resource (Knight et al., 2004). For 
digital educational resources, the listed author and the digital 
publisher are the only authorities for the resource quality 
(Cafolla, 2006). 

REVIEW MODELS FOR DIGITAL EDUCA¬ 
TIONAL MATERIALS IN THE GEOSCIENCES 

Various efforts have been made to develop review 
systems for the growing number of digital educational 
resources that allow a faster turnaround than the textbook 
review process. Ten prominent examples of review models in 
the geoscience education community are characterized in 
Table I. 

Seven of the collections listed in Table I were produced 
for or are accessible through the Digital Library for Earth 
System Education (DLESE), a collection focused on provid¬ 
ing digital learning resources for earth system science topics. 
DLESE comprises a small reviewed (1,076 resources) and a 
broad nonreviewed (12,960 resources) collection section 
(DLESE, 2012). The reviewed collection is only growing 
slowly as a result of a significant decrease in DLESE funding 
(Marlino et al., 2008; Fellows, 2009; Mervis, 2009b). 

MODEL FOR TRANSPARENT RIGOROUS 
PEER REVIEW FOR EDUCATIONAL RE¬ 
SOURCES 

CLEAN (www.cleanet.org) developed a peer-review 
process for educational resources by incorporating elements 
of the review models described previously (Table I) as well as 
from the scholarly peer-review process. The scope of the 
CLEAN collection is defined by the Essential Principles of 
Climate Literacy (USGCRP, 2009) and the internally 
developed energy awareness principles (CLEAN, 2012). 
With the publication of the Energy Literacy Principles 
(DOI, 2012), we are also aligning the collection to these 
nationally endorsed key ideas. Using this literacy-based 
approach, CLEAN is curating a collection that covers the 
breath of relevant online materials for the two subject areas. 
The CLEAN collection features open-access learning activ¬ 
ities, videos, short demonstrations/experiments, visualiza¬ 
tions, and modules/units geared toward secondary and 
postsecondary educators. 


CLEAN Review Process 

The review process is outlined in Fig. 2, and includes six 
steps: 1) informal triage during resource harvesting, 2) 
formal triage, 3) first and 4) second general review, 5) panel 
review, and 6) expert science review. Resources enter the 
review process either via addition by a CLEAN review team 
member or submission via a public form. The CLEAN review 
team attempts to search all major resource pools to harvest 
existing materials (step 1). Even though the CLEAN team 
has screened thousands of digital materials (see Fig. 3), not 
all the existing resources have been reviewed because new 
resources are continually being developed, partially as a 
result of recently increased funding opportunities in climate 
change education. 

Resources entered in the online review tool undergo a 
brief vetting (step 2) in which the relevance of the resource 
to the collection is assessed and an initial qualitative 
recommendation is given. Resources that pass the initial 
vetting are subject to two rounds of general review (steps 3 
and 4), which are documented in the online review tool (Fig. 
4). These two general reviews assess three aspects of the 
resource: scientific accuracy, pedagogic effectiveness, and 
usability/technical quality. The questionnaire prompts re¬ 
viewers to consider all aspects that characterize a high- 
quality resource, including best practices, in the design of 
educational resources. A unique set of questions was 
developed for each resource type (activity, video, visualiza¬ 
tion, demonstration/experiment, module/unit). Reviewers 
answer specific review questions on a Likert scale, provide 
comments on strengths and concerns in narrative form, and 
summarize in an overall rating for each of the three 
categories. The reviewer then gives an overall qualitative 
recommendation (low, medium, or high priority), which 
determines the path a resource takes through the review 
process. Resources rated to be of medium or high quality 
move forward to the next step, whereas any resource rated 
low priority does not move on. The CLEAN review team 
consists of six in-house reviewers, experienced middle 
school, high school, and college-level educators as well as 
scientists. 

Resources that pass the two general reviews are 
presented to a panel of four reviewers (step 5). The panels 
include at least one climate or energy scientist, experienced 
educators, and a CLEAN team reviewer. Modeled after the 
National Science Foundation panel review process, each 
reviewer is assigned a quarter of the panel resources to 
present during the review panel meeting for discussion. The 
panel discusses each resource, considering the initial reviews 
and adding their own insights. Finally, the panel recom¬ 
mends whether or not a resource should be included in the 
CLEAN collection. The panel also has the opportunity to 
forward a resource to the CLEAN editorial board for further 
discussion and final decisions in case some issues remain 
unresolved after the discussion. Bringing together educators 
and scientists in a panel ensures that attention is paid to 
both scientific accuracy and pedagogic effectiveness. All the 
reviewer comments are summarized as annotations (notes to 
the user), which are finalized by the panel if a resource 
passes and then provided on the public CLEAN page 
separately for each resource in the collection (Fig. 5). Review 
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TABLE I: Summary of ten prominent review processes developed in the geosciences. 



DLESE Community 

Review System 

DLESE Peer-Review System 
for Digital Geoscience 
Education Resources 

DLESE—Partnership 
Review System 

Collection audience 

Educators; learners at all levels 

Earth science instructors 

User of JESSE and UMADA 3 

Thematic focus of 
collection 

Geosciences 

Geosciences 

Geosciences 

Resource type reviewed 

Activities 

Variety of resource types, 
mainly activities 

Complex educational resources¬ 
like modules 

Review steps 

Community reviews, 4 specialist 
review, robustness review, editorial 
board, developer contact 

Reviews, 4 panel review, 
developer contact 

Reviews (completed online 
addressing parts of resource) 

Number of reviewers 

13+ 

2 plus panel 

3-6 

Expertise of reviewers 

Educators, scientists 

Geoscience and geoscience 
education experts 

Domain experts 

Written reviews 

Yes 

Yes 

Yes, online comments 

Classroom tests 

Yes 

No 

No 

Test of technical quality 

Yes 

Yes 

No, peer reviewers must be able 
to access and use it 

Review panels 

No 

Yes 

No (online group discussions) 

Public comments? 

Yes 

No 

No 

Iteration with developer 

Yes 

Yes 

Yes 

Reviewer comments posted 
online? 

Yes 

No 

No 

Star rating of resources 

No 

Yes 

No 

How to contribute 

Resources from DLESE broad 
collection, reviewed by community 
members 

Provided by selected 
collections 

Chosen by collection developer 

Number of reviewed 
resources in collection 

160 

155 

2 in JESSE 

Collection growing? 

No 

No 

No 

References 

Kastens, 2001, 2005; Kastens et al., 
2005; Arko et al., 2006; Kastens and 
Holzman, 2006 

Mayhew and Hall, 2007 

Weatherley et al., 2002 

Collection URL 

http://www.dlese.org/dds/histogram. 

do?group=subject&key=crs 

http://serc.carleton.edu/ 

introgeo/index.html 

Not longer available 


1 American Association of the Advancement of Science (AAAS)—Project 2061. 

2 The Climate Change collection (McCaffrey, 2009) was a pilot test for the CLEAN collection. 

3 JESSE—Journal of Earth System Science Education; UMADA—Unidata Meteorological Applications Discussion Area (data visualization software). 

4 Based on review criteria defined in Manduca and Mogk (2000). 

5 Science Educator's Guide to Selecting High Quality Instructional Materials. Project 2061 is not building a collection themselves but their review system was 
adopted by Phenomena and Representations for the Instruction of Science in Middle Schools (PRISM) project features a total of 324 digital middle school 
resources in 6 different topic areas, 134 of which are focused on earth science and ecology (PRISM, 2009). 

6 NASA Science Mission Directorate (SMD) education digital library is in development (planned release. Fall 2012). 

7 Under development. 


panels have been conducted during face-to-face meetings as 
well as virtually. 

The scientific accuracy of a resource is assessed during 
the general reviews. However, in order to ensure high 
scientific quality, CLEAN solicits an additional science 
review by a specialist in the field for each resource (step 


6). This expert scientist comments on the scientific content of 
the resource and recommends whether or not to include the 
resource in the CLEAN collection. Comments from the 
scientist are posted as annotations on the public resource 
page of accepted resources (Fig. 5). Each scientist is asked to 
keep the target grade level of the resource in mind when 
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TABLE I: Extended. 


MERLOT—Multimedia 
Educational Resource for 
Learning and Online Teaching 

NASA Earth and Space Science 
Education Product Review 

AAAS—Project 2061 1 

Climate Change Collection 2 

Higher education faculty and 
students 

Grades K-12, college, and 
informal educators 

Grades K-12 educators 

Grades 6-14 educators 

Geosciences and other subject 
areas 

Earth and space sciences 

Science 

Climate change 

Wide variety of learning resources 
(19 different types, e.g., activities, 
tutorials, simulations) 

All NASA educational resources 
(e.g., lesson plans, curriculum, 
exhibits, games, courses) 

Variety of instructional 
materials 

Variety of resources 

Triage, reviews, editorial board 
decision, developer contact 

Reviews, panel discussion, NASA 
forum and/or NASA consultation, 
developer contact 

Review of learning goals, of 
resource effectiveness, 
summary 

Reviews, 4 panel discussion 

2 plus editor 

5-7 

2 

2-3 

Faculty member of discipline 

Formal and informal educators, 
scientists 

K-12 educators 

Teachers, climatologist, 
cognitive scientist 

Yes 

Yes 

Yes 

Yes 

No 

No, most resources classroom- 
tested before submission 

No 

No 

No, peer reviewers must be able 
to access and use it 

No, peer reviewers must be able 
to access and use it 

No, peer reviewers must be 
able to access and use it 

No, peer reviewers must be 
able to access and use it 

No 

Yes 

No 

Yes 

Yes 

No 

No 

No 

Yes 

Yes 

Not available 

No 

Yes 

No 

Not available 

Yes 

Yes 

No 

Not available 

Yes 

User suggestion only 

Submitted by developer 

Not available 

Chosen by reviewers 

3,354 total (geoscience: 4) 

~800 

Not available 5 

40 

Yes 

Yes 

Not available 5 

No 

Malloy and Hanley, 2001; Cafolla, 
2006; http://taste.merlot.org/ 
peerreviewprocess.html 

http: //nasareviews. strategies. org/ 

http://www.project2061.org/ 

publications/ 

EducatorsGuide/online/ 

http://serc.carleton.edu/ 

climatechange/index.html 

www.merlot.org 

http://www.dlese.org/dds/ 
histogram.do?group=subject& 
key=eserev; http:// 
teachspacescience.org 

Not available 7 

http://serc.carleton.edu/ 

climatechange/summary.html 


judging its scientific quality because simplifications are often 
necessary for complex science topics in resources for 
younger students. The scientist also notes if a resource 
addresses cutting-edge science and needs to be revisited 
regularly to check for currency of the science content. The 
pedagogic effectiveness is assessed by master teachers and 
college-level faculty during the two general reviews and the 
review panel. 

CLEAN makes a concerted effort to engage with 
resource developers. Unlike a journal peer-review process 
where authors submit articles for peer review, the initial 
CLEAN collection primarily features digital resources that 
were identified by the CLEAN team members from a wide 


range of online resource pools (Fig. 2). The CLEAN team 
contacts developers of resources that have passed the full 
review process and provides them with a "CLEAN Selected" 
logo (Fig. 6) to place on their accepted resource. CLEAN 
now also offers developers the opportunity to choose an 
iterative review process. In this iterative process, the CLEAN 
team provides feedback from the reviewers to the developer 
of a resource through the review process. Developers 
address the comments and implement changes into the 
resource prior to acceptance in the collection. Developers of 
resources that fail the review process can resubmit the 
resources for review after reviewer concerns are addressed. 
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TABLE I: Extended. 



DWEL—Digital Water Library 
Project 

GLOBE—Global Learning and 
Observations to Benefit 
the Environment 

CLEAN—Climate Literacy and 
Energy Awareness Network 
Collection 

Collection audience 

Grades K-12 and informal 
educators 

Middle school and high school 
educators 

Grades 6-16 and informal 
educators 

Thematic focus of 
collection 

Science, economics, policy of 
water 

Earth system education 

Climate and energy science 

Resource type reviewed 

Variety of resources, mainly 
activities 

Variety of resources, mainly 
activities 

Activities, visualizations, videos, 
demonstrations, experiments, 
modules, units 

Review steps 

Vetting, resource type specific 
reviews 4 

Reviews, 4 input from GLOBE 
community, developer feedback, 
NASA product review 

Vetting, general reviews, review 
panel, expert science review, 
developer contact 6 

Number of reviewers 

2 

7 

5-7 

Expertise of reviewers 

K-12 and informal science 
educators 

Classroom teachers, educators, 
expert scientist 

Scientists, formal and informal 
educators 

Written reviews 

Yes 

Yes 

Yes 

Classroom tests 

No 

Yes 

No 

Test of technical quality 

No, peer reviewers must be able 
to access and use it 

Yes 

No, peer reviewers must be able 
to access and use it 

Review panels 

No 

No 

Yes 

Public comments? 

Yes 

No 

Yes 

Iteration with developer 

No 

Yes 

Yes 

Reviewer comments posted 
online? 

No 

No 

Yes 

Star rating of resources 

No 

No 

No 

How to contribute 

Teachers identify resources 

GLOBE-produced resources 

Submitted by developer, CLEAN 
team collected 

Number of reviewed 
resources in collection 

370 

>100 

480 

Collection growing? 

No 

Yes 

Yes 

References 

Moore and Aivazian, 2002; 
Aivazian et al., 2003; http:// 
www.dlese.org/Metadata/ 
collections/scopes/dwel-scope. 
php 

http://classic.globe.gov/fsl/html/ 

templ.cgi?dlesescope&lang= 

en&nav=0 

http://cleanet.org/clean/about/ 

review.html 

Collection URL 

http://www.dlese.org/dds/ 
histogram. do ? group= 
resourceType&key=dwel 

www.globe.gov 

http://cleanet.org/clean/ 

educational_resources/index.html 


Development of CLEAN Review Process and Criteria 

In the multistage CLEAN review process, both educa¬ 
tors and scientists judge the quality of an educational 
resource. The process builds on previously developed 
guidelines from the other review processes listed in Table I 
and pilot-tested in the climate change collection (McCaffrey, 
2009). The CLEAN process combines the strengths of the 
different efforts into a new rigorous review system. 


All review activities and comments are captured in a 
custom-built online tool, which has been refined over the 
first two project years. For any given resource, five to seven 
reviewers with different areas of expertise (master teachers, 
college-level faculty, and scientists) evaluate and provide 
comments throughout the process to ensure that only the 
best resources that are in scope with the collection pass 
through the system filters. The in-house CLEAN review 
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FIGURE 1: Peer-Review Process for Scientific Journal Articles. Author submits previously unpublished manuscript to 
a journal's editorial office. Editors decide on general fit of the manuscript and send it out to multiple reviewers. 
Reviewers are asked to evaluate the manuscript and return it to the editor with comments and recommendation 


about acceptance to the journal. Based on the reviews, the 
if revisions are necessary before publication. 

team completes the general reviews; the panel brings 
external reviewers together. The review process typically 
takes 3-5 months for a given resource. 

As mentioned above, we have developed a unique set of 
questions (review criteria) for each resource type that we 
review. An initial attempt was made to measure interre¬ 
viewer reliability with each of these review criteria. However, 
many questions are subjective (e.g., "Is the resource 
engaging and motivating for students?"), making it difficult 
to implement quantitative measures. Thus, all CLEAN 
review outcomes are qualitative. Participation of CLEAN 
team members in the development and refinement of the 


editor decides whether or not to publish the manuscript or 


review criteria and process has established a common 
understanding of each review instrument. The six-step 
CLEAN review process is supported by a custom-built 
technical infrastructure that documents and archives the 
process for each resource (Fig. 4). 

LESSONS LEARNED 

Definition of Collection Scope 

Review teams initially struggled to determine which 
resources should be included in the CLEAN collection. One 
problem we faced was that the literacy documents that define 
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t 




lommuni cation 


Analysis of Caps 
and Thin Spots 


FIGURE 2: CLEAN Review Process. An initial informal triage step (1) during resource harvesting is followed by a 
vetting step (2) that ensures that a resource is compatible with the collection scope; two separate written reviews (3 plus 
4) follow. A panel review (5) considers all previous reviews in a panel discussion before acceptance in the collection. A 
final quality check by an expert scientist marks the last hurdle for its inclusion in the CLEAN collection (6). 


the collection scope are knowledge statements and not a 
simple list of learning goals, and hence not an easy guide to 
finding relevant resources. Internal discussions and the 
development of a comprehensive set of climate and energy 
topic terms ("vocabularies") helped the team define the 
collection scope (CLEAN, 2011). The clearly defined collection 
scope has also allowed us to identify gaps and thin spots in 
the collection, which again have informed the CLEAN 
resource-collecting efforts as well as resource developers. 

Addressing Needs of Diverse Audiences 

CLEAN strives to support educators in teaching diverse 
audiences. However, the review team found it challenging to 
define the needs of a diverse audience and determine how to 
address this nebulous group in the review criteria question¬ 
naire because each of the individual sectors within the 
diverse audience has specific needs. Initially, CLEAN 
experimented with tagging resources that seemed to appeal 
to a specific underrepresented audience (such as English 
language learners. Native Americans, Hispanics), similar to 
the tagging implemented by other efforts (Kastens et al., 
2005). However, the results were not satisfying. Instead, the 
team added new review questions to focus the reviewers' 
attention on these aspects. Information for specific audiences 
is included in the annotations. 

Addressing the specific needs of different diverse 
audiences has proved to be prohibitively time consuming. 
If the task was to be done well, CLEAN would have to bring 


in experts for each diverse audience. Furthermore, most 
aspects that make a resource accessible to a diverse audience 
(e.g., using hands-on and inquiry-based instruction, adding 
visuals, limiting text) make it more accessible to all students. 

Challenge: Obtaining Expert Science Reviews 

As we have built the collection, it has proven difficult to 
obtain expert science reviews because the range of expertise 
required was daunting. Soliciting expert reviews is time 
consuming because each resource has to be matched 
individually with a specialist. Expert science reviews usually 
take between 20 and 60 minutes and are therefore less time 
consuming than typical journal reviews. CLEAN asks 
scientists to complete the reviews as part of a service to 
science education and does not offer compensation. Unlike 
the review of scientific papers, review of educational 
resources is not generally recognized as a merit for an 
academic career. We are still working on an efficient process 
of soliciting expert reviews, increasing our database of 
potential expert science reviewers, and matching experts 
with reviews without frequently turning to the same people. 
We have engaged scientists during professional meetings, 
which is a promising way of recruiting expert science 
reviews; however, this process still needs refinement. 

Communication with Resource Developers 

The CLEAN team is still experimenting with approaches 
for engaging resource developers. Resource developers have 
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1555 passed step 2: " ve " infl 
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1033 passed 
532 passed 


Step 4: 2 nd General Review 
Step 5: Panel Review Camp 


Step 6: Expert Science Review 

Passed 170 resources, Failed: 20 resources, Pending: 342 resources 
(Pending editorial board review: additional 107 resources) 


FIGURE 3: CLEAN Review Process Outcomes. Summary of CLEAN review outcomes for review period March 2012- 
May 2012. 


different levels of interest in and availability for the CLEAN 
process. Some developers are very engaged and want to be 
involved in the review process. Others have developed 
resources with the CLEAN review criteria in mind and are 
willing to update their resources based on feedback from 
CLEAN reviews. Some resources we reviewed have been 
developed and published under funding that has expired or 
the resource developers have moved on. These developers 
are not typically available for a dialogue with the CLEAN 
team and often lack funding to implement suggested 
changes. Because of the variety of availability on the part 
of the developers, the level of communication with each one 
varies. Ideally, the CLEAN review process will evolve into a 
submission peer-review process similar to a journal article 
submission, where developers are willing to engage in an 
iterative review process to improve their resource in order to 
have it accepted into the CLEAN collection. 

Dynamic Digital Environment 

One challenge faced by CLEAN and other reviewed 
collections is the dynamic landscape of digital resources— 
new resources are constantly being created and old ones 
removed. The removal of resources is particularly problem¬ 
atic for CLEAN because we do not host any of the resources. 
We provide information about the resource in our annota¬ 
tions and the direct URL to the resource. In addition to 
disappearing, resources sometimes deteriorate or evolve 
(Weller, 2000; Kastens et al., 2005). Automatic link-checking 
programs facilitate repair of broken links directed to the 
resource. However, they do not address broken links or 
other deterioration within the resource. To address the 
challenge, we note the dates the resource was developed and 
last updated during the review. We currently do not have a 


mechanism to flag resources that were changed after the 
review. 

Scalability of Review Process 

The CLEAN review process was developed for a small 
collection with a clearly defined thematic scope. The staff¬ 
intensive CLEAN review process was not designed to be 
scalable to a large collection that adds several thousand new 
catalog records annually. However, moving from resource¬ 
intensive face-to-face review camps to virtual camps reduced 
the necessary funds for the review process. In this sense, the 
review process is very much like the peer-review process of a 
journal that conducts reviews of only a few hundred 
manuscripts per year. Review processes for much larger 
collections of educational materials have to be simpler and 
cannot achieve the same rigorous level of curation as CLEAN. 

Sustainability 

The CLEAN collection and the processes and tools used 
to build it have been developed with the needs of the 
broader community—ranging from the funding agencies to 
individual educators—in mind. The longer-term sustainabil¬ 
ity of CLEAN is dependent on it continuing to meet the 
needs of this community. We have defined what is needed to 
maintain at a minimum level CLEAN core activities and 
several tiers of activities that would more fully develop the 
CLEAN core and would expand on and leverage the CLEAN 
collection and services. This has helped us define the various 
avenues through which we can develop partnerships to fund 
the core activities, develop proposals to extend on CLEAN 
core activities, and provide review services to groups who are 
developing materials. We are currently pursuing these 
partnerships to develop lines of continuing funding. 
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FIGURE 4: Screenshot of CLEAN Review Tool. Custom-built review database in which all review information is 
stored separately for each resource. All the reviews, comments, and ratings are digitally available. 


CONCLUSION 

Easy and inexpensive digital publishing leads to a vastly 
growing number of digital educational materials, often with 
unclear credibility, making it difficult for educators to 
evaluate the quality. Reviewed collections of digital educa¬ 
tional materials build trust in educators and minimize the 
time educators have to spend searching for high-quality and 
relevant materials. 

The scholarly journal peer-review process as well as the 
formal textbook adoption processes are established models 
that have informed the development of review processes 
geared toward digitally available teaching resources. One 
key difference between reviewing digital educational re¬ 
sources and the scientific journal peer-review process is that 
journal articles are published in the media owned by the 
company leading the review process and only after the 
author successfully completes the review process for the 
article. On the other hand, digital educational resources are 
usually published on the Web site of the developer. Whether 
or not they pass a review process does not affect the digital 


publication itself. The other key difference is that multiple 
reviewers (scientists and educators) are needed to evaluate 
the quality of educational resources. 

The CLEAN project has developed a set of resource- 
type-focused review criteria and a robust peer-review 
process to evaluate educational resources by incorporating 
aspects of previously developed educational review process¬ 
es, by addressing the specific needs for educational 
materials, and by following the established scientific peer- 
review process in bringing scientists and educators together 
to review resources using a panel. Educators of middle 
schools, high schools, and colleges alike are excited about 
the content of the CLEAN collection (Table II), specifically 
because both educators and scientists have reviewed the 
resources thoroughly. CLEAN provides what they need to 
teach about climate and energy topics at different instruc¬ 
tional levels and the rigorous curation reduces the time they 
have to spend looking for effective and accurate resources. 

CLEAN uses a literacy-based approach to curate a well- 
rounded collection containing resources to teach all the key 
ideas as laid out by the climate literacy principles and the 







































































J. Geosci. Educ. 60, 295-308 (2012) 


Peer-Review of Digital Educational Resources 305 


Using Satellite Images to Understand Earth's Atmosphere 

Todd Ensign, et. al.. Earth Exploration Toolbook, TERC 

In this Earth Exploration Toolbook chapter, students select, explore, and analyze satellite imagery. 
They do so in the context of a case study of the origins of atmospheric carbon monoxide and 
aerosols, tiny solid airborne particles such as smoke from forest fires and dust from desert wind 
storms. They use the software tool ImageJ to animate a year of monthly Images of aerosol data and 
then compare the animation to one created for monthly images of carbon monoxide data. Students 
select, explore and analyze satellite imagery using NASA Earth Observatory (NEO) satellite data and 
NEO Image Composite Explorer (ICE) tool to investigate seasonal and geographic patterns and 
variations in concentration of CO and aerosois in the atmosphere. 

Activity takes about three to four class periods. Computer and Internet access required. 


• Introduce students to ImageJ before beginning the exercise. 

• Instructors should emphasize the effect of aerosols on the energy budget and climate. 

• Suggest pairing students for activity 

• Excellent use and appi'cat'on of science from remote sensing data with a problem-based scenano 

• Makes use of the type of tools scientists use to analyze satellite images to study change. 

• Hands-on exercise uses NEO data to explore carbon monoxide concentrations and aerosols. 

• Good organization of learning materials and a log cal learning sequence eads students to analysis and 
understanding of complex data sets. 

• Instructions are clear, thorough, and well-written. 

• The case study provides good background information for students (and teachers). 

• Excellent going further activities for advanced students 

• Hands-on exercise uses NEO data to explore carbon monoxide concentrations and aerosois. 

• This resource engages students in using scientific data. 

See other dau-rich activities 

• The 'show me' feature that helps students through problems and procedures using software is just 
exemplary and allows students to check their progress frequently throughout the activity. 

FIGURE 5: Example of CLEAN Public Catalog Record. CLEAN collection hosts a Web page (catalog record) per 
resource with resource URL, content summary, a summary of reviewer comments (annotations), and teaching tips, 
along with relevant and searchable metadata such as appropriate grade level, standard and benchmark alignment, 
and relevant benchmark maps. 
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CLEAN 


FIGURE 6: CLEAN Selected Logo. Quality stamp that 
can be placed on the original resource Web site by the 
developer to indicate that resource passed the CLEAN 
review process. 

energy literacy principles. This literacy-based approach 
facilitates self-assessment of how well the collection 
addresses primary topics and ideas at different grade levels 
and allows identification of gaps in the collection with 
respect to addressing all the climate literacy and energy 
literacy essential principles and fundamental concepts. 

Services that CLEAN offers to resource developers 
include review of resources, dissemination of geoscience 
educational resources beyond the originally intended audi¬ 
ence, and analysis of the gaps in the collection's resources as 
well as guidance on what makes a high-quality educational 
resource. The investment in the CLEAN review process has 
built a community of diverse experts who can now share 
their resources and expertise learned through participation 
in the CLEAN review process with the broader community. 
The CLEAN review process can be adapted for other 
thematic collections because most review questions identify 


high-quality teaching materials and are not thematically 
focused on climate and energy science. 
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